A software and hosting optimization for an international law firm

FMLA Edge Overview

FMLA Edge is a python web app built with django, an MVC framework. Edge has various dependencies on hosting services such as sftp, database, and redis. Edge also heavily depends on background tasks to accomplish much of its functionality. These tasks often took much too long to complete for key high-profile clients (sometimes 8 to 10 hours).

Ogletree hired BoltAffect to transition FMLA Edge from its former hosting provider and correct some of the architectural issues that lead to the performance problems.

Initial hosting architecture and cost

FMLA Edge was hosted with two “web” servers that hosted both the website and the background tasks that did most of the data processing. Additionally, the first “web” server in the pair hosted an FTP server, Redis message broker, import file storage, and a few other services. This monolithic hosting environment tended towards memory starvation when background task processing was high.

Generally the hosting looked like this:

Unfortunately FMLA Edge was written with the above hosting configuration in mind. Sections of the code (both for the web site and background tasks) utilized the fact that most of the system ran on a single server. File paths to specific files on disk were coded right into the source code and server permissions were configured to allow access to various resources on a single machine. For instance, the python project was run under the FTP user so that the python project could read files delivered by FTP. This required the FTP user to have execution permissions: a terrible security blunder.

To scale up the system the only option was to grow the size of the servers. This is known as Vertical Scaling. You take underperforming servers and just make them more powerful. This was especially costly.

From vertical to horizontal hosting

BoltAffect migrated FMLA Edge to new hosting where individual machines were responsible for single tasks, commonly called Service Oriented Architecture (SOA). Web servers are only responsible for the web front end, and dedicated app servers run the background tasks. Similarly sftp, databases, and other services were all moved off to their own machines.

This dramatically grew the number of servers, but each server could be sized and optimized for the workload and more servers to handle specific tasks can easily be added as more customers are added:

Rewriting sections of the application to transition it from a monolithic hosting architecture to a Service Oriented Architecture was important for three major reasons:

1. Performance: the old hosting environment had a problem with resource starvation (specifically around memory)
2. Cost: Vertical scaling is the only option with monolithic hosting, whereas SOA supports horizontal scaling. Horizontal scaling is often much less expensive than vertical scaling.
3. Security: isolated services are easier to lock down.

Say no to polling

Besides moving to a SOA, BoltAffect identified multiple places in the background task system where database polling was implemented. Background tasks would run and check if certain conditions were met and then kick off additional tasks to complete work based on those conditions. Those polling tasks were replaced by an event driven system where user or import events would directly kick off the background tasks.

Any user generated event was then placed into a priority queue so that the user would see nearly instantaneous results.

Proof of the pudding is in the eating

Containers?

It’s worth noting that hosting technologies such as LXC containers (and Docker to manage them) can be used to help manage deployments of software that subscribe to a service oriented architecture. However, the addition of those tools is not necessarily needed and comes with a performance cost. This project was implemented with Virtual Private Servers without the use of containerization and the resulting complexities that containers would have added to this project. Furthermore, the addition of containers prior to the software updates needed to implement SOA would have degraded performance. Containerization is not a silver bullet to poor performance.

As the number of VPS’s increases Ogletree may revisit containers as a management solution once larger scale is needed.

Micro-services?

The updates to the code base did NOT transform the project to a set of smaller projects that would normally be considered micro-services. Even though micro-services are trending these days, a monolithic project with good internal design can be easier to understand and work on. Each app of a django project can be considered a “micro-service”. So deploying the monolithic codebase as different services in the hosting environment is not only possible but eliminates large scale software rewrites. This has proven to yield the best of a monolithic software project and service oriented hosting.

Conclusion

This project took considerable effort to complete since the original codebase had many hosting assumptions baked in. However, both the cost and performance outcomes made this project extremely successful. Ogletree slashed the hosting cost and gained significant performance gains.